Of goals and habits.

In the television series TheWire, addicts Bubbles and Johnny regularly engage in bizarre and elaborate schemes to obtain drugs, ranging from robbing an ambulance to stealing drugs by lowering a fishing line from a rooftop. That these fictional crimes can be both so meticulously planned and yet focused on such narrow, shortsighted goals highlights a gap in our understanding of how the real brain deploys deliberative vs. automatic mechanisms to make decisions. On a standard account, people can deliberatively evaluate the consequences of candidate actions, which gives us our flexibility to dream up novel plans. Alternatively, the brain can crystallize repeatedly successful behaviors into habits: learned reflexes that free up resources by executing the behaviors automatically (although at the expense of inflexibility and, it is believed, underpinning pathological compulsions). As with most dichotomies, the problem with this view is that the world is not so black and white. Much as the drug-seeking behavior of addicts seems not to fit into either category, for healthy behaviors also, neither of these two sorts of decision making is very practical on its own. In PNAS, Cushman and Morris suggest a hybrid of these mechanisms, and show behavioral evidence that humans use it to plan actions (1). The study of Cushman and Morris (1) draws on recent advances using computational models of learning to make these strategies explicit enough that their hallmarks can be measured in choice behavior. Decisions are often modeled as determined by one’s estimates of the rewards expected from different options. There are many different methods to compute these decision variables. Deliberative planning can be formalized by an algorithmic family, called “model-based reinforcement learning” (2), which evaluates candidate sequences of actions much like a chess computer does, by exhaustively searching the “tree” of their future consequences, generated using a learned model of the task contingencies (like the rules of chess or the map of a maze) (Fig. 1A). The key feature of habits, in this view, is that they instead rely on a simpler summary of the end results of this computation, such as the overall long-run reward expected following some action (2). This precomputation gives them both their simplicity and inflexibility. These summaries do not actually need to be derived from exhaustive computations using a model, but instead can be learned directly—although slowly—“modelfree” from experience (3). Cushman and Morris (1) propose a hybrid of these strategies, in which each system simplifies the problem faced by the other (Fig. 1B). The hypothesis is that, in a multistep decision problem, model-free learning selects a goal or subgoal, then model-based planning figures out how to get there. If I wanted to travel to Paris, a goal might be the airport; in chess, it might be forking the opponent’s queen. This is a useful division of labor because, in general, finding the best action using a model-based search is too complicated because of the many possible trajectories of future action. However, given a single goal, finding a good path to it can be more manageable. Meanwhile, although model-free learning is computationally simple, it learns slowly in large, multistep tasks. Narrowing its attention to a smaller, more abstracted problem—choice between a few candidate goals— lets its simplicity shine. The key to the experiments of Cushman and Morris (1) is that these strategies learn in different ways from experience in trial-anderror decision tasks (4). Indeed, just how behavior changes following a single, carefully arranged trial of experience can reveal things like whether a goal was updated, and if so how. The authors (1) harnessed the efficiency of online data collection to zero-in on these rare informative choices across a large number of subjects. In the studies (1), participants chose between actions, which led to intermediate situations represented by different colors, and then to monetary reward. If a choice that led to the blue intermediate goal was followed by a large pay-off, subjects tended to choose actions leading to blue again later. This was true even if the later action needed to get to blue was a different or even completely novel one, which suggests that planning to get to blue was model-based. This is because model-based planning, by evaluating actions in terms of their predicted consequences, can generalize across different ways to achieve the same result (5). If I want to get to the airport, model-based learning can find a route there from anywhere, whereas model-free learning can only repeat previously successful routes. Surprisingly, a rewarded choice for blue was repeated even if the subjects in the study (1) had failed to achieve the intended blue goal on the initial choice, so that this reward was irrelevant to the true value of the goal. This apparent mistake is exactly what modelfree selection predicts, because it assesses the success of a strategy (here, choosing blue) in terms of the reward received. Four experiments $$ $ ¢